deploy: Change BootLoaderSpec filenames so they can be used for sorting
Currently the BLS snippets are named ostree-$ID-$VARIANT_ID-$index.conf,
but the BLS config files are actually sorted by using the version field
which is the inverse of the index.
In most places, _ostree_sysroot_read_boot_loader_configs() is used to
get the BLS files and this function already returns them sorted by the
version field. The only place where the index trailing number is used is
in the ostree-grub-generator script that lists the BLS files to populate
the grub config file.
But for some bootloaders the BLS filename is the criteria for sorting by
taking the filename as a string version. So on these bootloaders the BLS
entries will be listed in the reverse order.
To avoid that, change the BLS snippets filename to have the version field
instead of the index and also to have the version before deployment name.
Make the filenames to be of the form ostree-$version-$ID-$VARIANT_ID.conf
so the version is before the deployment name.
Signed-off-by: Javier Martinez Canillas <javierm@redhat.com> Closes: #1654
Approved by: cgwalters
Simon McVittie [Sun, 24 Jun 2018 11:56:49 +0000 (12:56 +0100)]
OstreeRepoFinderConfig: Fix guint/gsize confusion
If a function has a guint "out argument", passing a pointer to a gsize
is not, in general, valid. On an ILP64 platform there is no problem
since guint and gsize are identical, but on an LP64 platform it will
overwrite only the first word of the gsize, leaving the second word
unaffected. On little-endian machines, if the second word is
zero-initialized (as it is here), the result is numerically equal to
the guint, but on big-endian machines the result is around 4 billion
times what it should be, resulting in
ostree_repo_finder_config_resolve_async() reading past the end of
the array and causing undefined behaviour.
In practice this caused assertion failures (and consequently test
failures) on Debian's s390x (z/Architecture), ppc64 (64-bit PowerPC)
and sparc64 (64-bit SPARC) ports.
ostree-grub-generator: sort BLS files by version instead of alphabetically
The ostree-grub-generator populates the grub.cfg menu entries using the
BLS config files. But it uses the ls command that by default sorts the
entries alphabetically, so the order won't be correct if there are more
than 10 deployments, i.e:
$ ls -1 /boot/loader/entries/
ostree-fedora-workstation-0.conf
ostree-fedora-workstation-10.conf
ostree-fedora-workstation-1.conf
...
So instead the -v option should be used to make ls use version sorting:
$ ls -1 -v /boot/loader/entries/
ostree-fedora-workstation-0.conf
ostree-fedora-workstation-1.conf
...
ostree-fedora-workstation-10.conf
Signed-off-by: Javier Martinez Canillas <javierm@redhat.com> Closes: #1653
Approved by: cgwalters
William Manley [Fri, 22 Jun 2018 10:37:37 +0000 (11:37 +0100)]
OstreeMutableTree: Document each private member of `OstreeMutableTree`
A prelude to my understanding. Unfortunately `OstreeMutableTree` provides
little encapsulation, as each member has setters† so it's difficult to come
up with a list of invariants.
† `files` and `subdirs` only have getters, but the getters return mutable
references to the internals, so we still can't reason about invariants.
Simon McVittie [Fri, 22 Jun 2018 00:10:55 +0000 (01:10 +0100)]
avahi: Fail immediately if we can't talk to D-Bus or Avahi
We special-case AVAHI_ERR_NO_DAEMON to not cause warnings, but if
we pass AVAHI_CLIENT_NO_FAIL to avahi_client_new, we never actually
see AVAHI_ERR_NO_DAEMON. Instead, we will get AVAHI_ERR_BAD_STATE
when we try to use the client.
Umang Jain [Tue, 19 Jun 2018 20:52:12 +0000 (02:22 +0530)]
lib/repo: Enforce min-free-space-* size check for regfiles in deltas
During the pull, there is an explicit check for free space on disk
vs. the size of uncompressed delta; But while writing the new content
objects that are generated, they have to honor min-free-space-* checks
too. We enforce this check in _bare_content_commit as that is where
we can know the final size of the new content object.
Colin Walters [Mon, 18 Jun 2018 14:47:49 +0000 (14:47 +0000)]
lib/repo: Do free space math under lock in error path
We were referencing the txn bits outside of the lock in the error
path. Generally shouldn't matter, but e.g. Rust wouldn't let us do this, and
race detector tooling will warn about it.
Colin Walters [Fri, 15 Jun 2018 21:10:14 +0000 (21:10 +0000)]
lib/archive: Tell g-ir-scanner to ignore the private libarchive bits
Squashes this warning:
```
src/libostree/ostree-libarchive-private.h:46: syntax error, unexpected typedef-name in ' g_autoptr(OtAutoArchiveRead) a = archive_read_new ();' at 'OtAutoArchiveRead'
```
Dan Nicholson [Sat, 25 Jul 2015 16:51:14 +0000 (09:51 -0700)]
deploy: Delete .updated file from /etc and /var on new deployments
Systemd units using ConditionNeedsUpdate run if the mtime of .updated in
the specified directory is newer than /usr. Since /usr has an mtime of
0, there's no way to have an older .updated file. Systemd units
typically specify ConditionNeedsUpdate=/etc or ConditionNeedsUpdate=/var
to support stateless systems like ostree.
Remove the file from the new deployment's /etc and the OS's /var
regardless of where they came from to ensure that these systemd units
run when booting new deployments. This will provide a method to run
services only on upgrade.
Matthew Leeds [Thu, 14 Jun 2018 21:33:10 +0000 (14:33 -0700)]
lib/prune: Don't modify dirent->d_name in place
Currently when I run `ostree prune` it hits a seg fault when the
hash_func is used (in this case g_str_hash) from the call stack
_ostree_repo_prune_tmp() -> g_hash_table_contains() ->
g_hash_table_lookup_node(). So the key, in this case dent->d_name, must
be corrupt in some way.
glnx_dirfd_iterator_next_dent() uses readdir() to get the dirent struct.
And according to the man page for readdir(3), "POSIX.1 explicitly notes
that this field should not be used as an lvalue" (in reference to
d_name). So this commit avoids modifying d_name in place and copies it
instead. This seems to avoid the seg fault.
Philip Withnall [Thu, 14 Jun 2018 15:47:43 +0000 (16:47 +0100)]
lib/repo-commit: Delay propagation of errors from abort_transaction()
If there’s a problem while aborting a transaction, store the error but
don’t report it until the end of the function — do a best effort at
clearing the rest of the transaction state first (since most of it
cannot fail).
If cleanup_tmpdir() fails (which, arguably, should not be a
showstopper), this allows a caller to recover and start a new
transaction in future.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1626
Approved by: jlebon
Umang Jain [Mon, 11 Jun 2018 21:06:31 +0000 (02:36 +0530)]
lib/repo: Add min-free-space-size option
Similar to min-free-space-percent but it supports specific sizes
(in MB, GB or TB). Also, making min-free-space-percent and -size
mutually exclusive.
min-free-space-percent does not give a fine tuning of the free disk
space that a user might decide to keep. It can translate to very large
size (e.g. 1% = ~10GB on 1TB HDD) or very small (e.g. 1% = ~330MB on 32GB
system like Endless devices). Hence, it makes sense to introduce a config
option to honor specific size as per the user.
Matthew Leeds [Tue, 12 Jun 2018 02:09:28 +0000 (19:09 -0700)]
admin: Fix list of subcommands in help and manpage
This adds subcommands that were missing from the ostree-admin man page,
and makes cosmetic fixes there and in the --help output to ensure
alphabetical order and remove trailing whitespace.
Jonathan Lebon [Thu, 7 Jun 2018 18:42:42 +0000 (14:42 -0400)]
switchroot: Allow letting ostree-prepare-root mount /var
In some scenarios, it might make sense to let `ostree-prepare-root` do
the `/var` mount from the state root as before. For example, one may
want to do some system configuration before the switch root. This of
course comes at the expense of supporting `/var` as a mount point in
`/etc/fstab`.
Matthew Leeds [Tue, 22 May 2018 22:55:14 +0000 (15:55 -0700)]
Make P2P API public (no longer experimental)
Currently the API that allows P2P operations (e.g. pulling an ostree ref
from a LAN or USB source) is hidden behind the configure flag
--enable-experimental-api. This commit makes the API public and makes
that flag essentially a no-op (leaving it in place in case we want to
use it again in the future). The P2P API has been tested over the last
several months and proven to work.
This means that since we're no longer using the "experimental" feature
flag, P2P builds of Flatpak will fail when using versions of OSTree from
this commit onwards, until Flatpak is patched in the near future. If you
want to build Flatpak < 0.11.8 with P2P enabled and link against OSTree
2018.6, you'll have to patch Flatpak. However, since Flatpak won't yet
have a hard dependency on OSTree 2018.6, it needs a new way to determine
if the P2P API in OSTree is available, so this commit adds a "p2p"
feature flag. This way the feature set is more semantically correct than
if we had continued to use the "experimental" feature flag.
In addition to making the P2P API public, this commit makes the P2P unit
tests run by default, removes the f27-experimental CI instance that's no
longer needed, changes a few man pages to reflect the changes, and
updates the bash completion script to accept the new commands and
options.
Matthew Leeds [Thu, 31 May 2018 00:20:08 +0000 (17:20 -0700)]
lib: Fix building against old glib versions
This commit includes libglnx.h in ostree-autocleanups.h, so we get the
g_autoptr backports wherever they're needed. Also, remove the "#include
libglnx.h" lines elsewhere that are no longer needed.
Philip Withnall [Mon, 28 May 2018 16:06:46 +0000 (17:06 +0100)]
lib/repo-pull: Support retries for delta superblocks
Use the recently introduced architecture for retrying network requests
on transient failure to do the same for delta superblock requests, now
that they’re queued.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1600
Approved by: jlebon
Philip Withnall [Mon, 28 May 2018 15:55:05 +0000 (16:55 +0100)]
lib/repo-pull: Support queuing delta superblock requests
Just like all the other requests made for delta parts and objects by the
pull code, use a queue for delta superblocks. Currently this doesn’t do
any prioritisation or retries after transient failures, but it could do
in future.
This means that delta superblocks are now subject to the parallel
request limit in the fetcher, which was a problem highlighted here:
https://github.com/ostreedev/ostree/pull/1453#discussion_r168321706.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1600
Approved by: jlebon
Philip Withnall [Wed, 30 May 2018 11:20:49 +0000 (12:20 +0100)]
lib/fetcher: Factor out HTTP status code handling from soup and curl
Use the same G_IO_ERROR_* values for HTTP status codes in both fetchers.
The libsoup fetcher still handles a few more internal error codes than
the libcurl one; this could be built on in future.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1594
Approved by: jlebon
Philip Withnall [Mon, 28 May 2018 15:25:53 +0000 (16:25 +0100)]
tests: Test pull behaviour when network timeouts occur
Extend test-pull-repeated.sh to test error 408 as well as error 500, to
ensure that the new retry-on-network-timeout code in ostree-repo-pull.c
correctly retries.
Rather than the 200 iterations needed for the error 500 tests, only do 5
iterations. The pull code internally does 5 retries (by default), which
means a full iteration count of 25. That seems to be sufficient to make
the tests reliably pass, in my testing — we can always bump it up to 200
/ 5 = 40 in future if needed (to put it in parity with the error 500
tests).
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1594
Approved by: jlebon
Philip Withnall [Mon, 28 May 2018 15:24:33 +0000 (16:24 +0100)]
ostree/trivial-httpd: Add --random-408s command line option
This is exactly like the --random-500s option, except that it will cause
error 408 (request timeout) to be returned, rather than error 500
(internal server error).
This will be used in a following commit to test pull behaviour when
timeouts occur.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1594
Approved by: jlebon
Philip Withnall [Fri, 25 May 2018 14:49:18 +0000 (15:49 +0100)]
lib/fetcher-soup: Map more SoupStatus codes to known GIOErrors
This allows the retry code in ostree-repo-pull.c to recover from (for
example) timeouts at the libsoup layer in the stack, as well as from the
GSocket layer in the stack.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1594
Approved by: jlebon
Philip Withnall [Tue, 22 May 2018 11:21:45 +0000 (12:21 +0100)]
lib/repo-pull: Support retrying requests on transient network errors
Allow network requests to be re-queued if they failed with a transient
error, such as a socket timeout. Retry each request up to a limit
(default: 5), and only then fail the entire pull and propagate the error
to the caller.
Add a new ostree_repo_pull_with_options() option, n-network-retries, to
control the number of retries (including setting it back to the old
default of 0, if the caller wants).
Currently, retries are not supported for FetchDeltaSuperData requests,
as they are not queued. Once they are queued, adding support for retries
should be trivial. A FIXME comment has been left for this.
Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #1594
Approved by: jlebon
Matthew Leeds [Tue, 29 May 2018 01:03:42 +0000 (18:03 -0700)]
Fix building P2P code against old glib versions
When building the OstreeBloom code against old versions of glib, we have
to have the libglnx headers included so that it defines
G_DEFINE_AUTOPTR_CLEANUP_FUNC and friends for us.
This is similarly true for test-repo-finder-mount.c which indirectly
includes ostree-autocleanups.h.
Matthew Leeds [Fri, 25 May 2018 22:46:34 +0000 (15:46 -0700)]
Allow compiling with experimental API but without curl/soup
This commit rearranges a few things in ostree-repo-pull.c so that OSTree
will successfully compile with experimental API enabled and without
libsoup, libcurl, or avahi:
1. Move ostree_repo_resolve_keyring_for_collection() so it can be used
even without libsoup or libcurl.
2. Add stub functions for ostree_repo_find_remotes_async() and
ostree_repo_pull_from_remotes_async(), and their _finish() counterparts,
so they return an error when libsoup or libcurl isn't available.
Jonathan Lebon [Mon, 28 May 2018 19:21:19 +0000 (15:21 -0400)]
ci: Bump to f28, except flatpak
We successfully switched to F28 in rpm-ostree. Time to follow suit here.
We exclude flatpak for now since we're still hitting test issues there
on F28.
Stopping on the first error is nice if you just want to know if everything is ok, but
if you want to figure out all that is wrong its nice to be able to continue and
print all corruptions.
Matthew Leeds [Thu, 24 May 2018 07:08:57 +0000 (00:08 -0700)]
bash-completion: Don't add a space after files and directories
Currently if ostree is completing a file or directory for you it adds a
space to the end, but this is an inconvenience when it's an intermediate
directory in a tree. It's better to let the user add the space after the
final directory, so this commit changes the bash completion to avoid
adding a space when completing files or directories.
Colin Walters [Mon, 30 Apr 2018 15:19:17 +0000 (11:19 -0400)]
lib/sysroot: Add wrapper API to prune system repository
The initial motivation for this is that the "staging" code currently
didn't rewrite the deployment refs, meaning that the staged commit
could be pruned.
Hence first, this new API ensures that deployments also
hold a strong ref to their commit, without relying on the magical
"deployment refs" that we inject. That has always been a weird
artifact of the strict layering separation between OstreeSysroot
and OstreeRepo.
I also plan to change rpm-ostree to start using this API to
hold references to base layers for client-side layering; it also
today generates various refs.
That said, if we still want to support multiple processes
writing to a single repo (as happens on EndlessOS today) we
still need to write refs; perhaps later we could add a concept
of "generators" or something that create refs based on whatever
logic?
Another minor thing this fixes is that we had a printf inside
the library; this propagates the pruned data to the higher level
which can log however it likes.
Colin Walters [Mon, 30 Apr 2018 14:31:33 +0000 (14:31 +0000)]
lib: Add a public helper method for pruning to find all ref'd commits
Prep for reworking how we do sysroot cleanup. We're going to
start doing more lowlevel pruning work there, and I wanted to avoid
duplicating the ref enumeration.
Andrea Galbusera [Tue, 15 May 2018 06:29:28 +0000 (08:29 +0200)]
u-boot: add support for devicetree
Likewise the corresponding support for syslinux introduced by commit c5112c25e4519835c4cd53f4350c1b2f2a477746, this one enables writing devicetree
filename into the uEnv.txt environment file for u-boot.
Since u-boot does not strictly defines variable names, here 'fdt_file' was
chosen as it appear to be one the most frequently adopted names in u-boot
default environments. Outer boot logic should of course comply with this choice
and use $fdt_file as the device tree file name to pass to boot commands.
This was tested on a custom board booting with u-boot.
Jonathan Lebon [Fri, 11 May 2018 19:03:58 +0000 (15:03 -0400)]
ci: Add .papr-ex.yaml
I'd like to start working towards migrating the OSTree CI towards the
new PAPR model running in PACI. In order to take it for a test drive
(pun intended) without yet replacing our current CI, the new PAPR now
knows to look for a `.papr-ex.yaml` file. So as to not conflict with the
current PAPR, no GitHub statuses will be output other than a
`required-ex` label, which plays the same role as `required`. For more
information, see:
The `.papr-ex.yaml` version is essentially the same except for:
1. the `FAH27-insttests` suite runs directly in the scheduled container
(notice the new `kvm: true` key)
2. the `host:` based contexts (rpm-ostree and flatpak) are missing since
we don't support provisioning a host just yet
Keeping them in sync could get annoying, though building confidence in
the new approach will allow us to migrate faster.
I feel like I'm drowning in a pile of experimental-but-almost-stable
features...
Anyways, since we made the feature opt-in in rpm-ostree in
https://github.com/projectatomic/rpm-ostree/pull/1352
let's mirror that a bit here with an environment variable so people
can play with it more easily.
The tests needed some tweaks; specifically we need to reload the
status fact after making changes. I'm still a bit uncertain
about the Ansible-as-tests.
But we add an upgrade test that uses the new environment variable.
Jonathan Lebon [Mon, 7 May 2018 13:41:27 +0000 (09:41 -0400)]
docs: Add "Hello World" example
Let's get practical faster in the manual and have a simple "Hello World"
example right off the bat to hopefully make it easier to grok how OSTree
works.
Also some minor tweaks on wording around comparisons to git.
Tobias Mueller [Sat, 5 May 2018 20:00:06 +0000 (22:00 +0200)]
repo: handle GPG_ERR_AMBIGUOUS_NAME in sign_data
This should give a more insightful error message if the user provides
a UID which is present on multiple keys.
This happens if you have an old key in your keyring which you are not
actively using any more, e.g. because it is too old. You still have
your old keys in your keyring, because you want to read old email
encrypted for that key, though.
The gpgme function used by ostree right now complains if a UID is found
on multiple keys:
https://www.gnupg.org/documentation/manuals/gpgme/Listing-Keys.html#index-gpgme_005fget_005fkey
The used API is too simple for that use case.
Note that it would be nicer if ostree picked the only valid signing key out
of the available keys rather than using the simplistic gpgme_get_key
function. It be nicer, of course, if there was such a gpgme function.
Colin Walters [Fri, 4 May 2018 17:51:13 +0000 (13:51 -0400)]
tests: Fix installed tests more
OK so I noticed that something was failing and we were missing
`set -xeuo pipefail` in our shells. That of course revealed
the ansible tests didn't actually work - my only defense
here is spending so much time fighting to get it through CI
and trying something new.
Anyways, to make the staged-deploy tests work we need a task
that actually uses `rpm-ostree override` rather than `usroverlay`.
Let's make this a bit saner and have a clean split between
tests that are "shell-script+usroverlay" and "ansible+override".
Colin Walters [Wed, 2 May 2018 15:59:21 +0000 (11:59 -0400)]
deploy: Use fdatasync() for new kernel/initramfs by default
While we do a `syncfs()` plus `FIFREEZE/THAW` for `/boot`, that
only comes during deployment finalization.
The code here today generally assumes that if the file exists
it's been fully written. So let's do a `fdatasync()` before
we do the `rename()`.
This just came out of looking through the code while working
on deployment staging. In that scenario there's a much larger
window between when we copy the kernel/initramfs and when we
sync `/boot`.
Colin Walters [Wed, 2 May 2018 16:17:20 +0000 (12:17 -0400)]
lib: Use `Locking:` term in docs
This is easier to `git grep` etc. versus ad-hoc English. Although
we still have some English for the prepare_transaction/commit which
acquire/release in separate phases.
Colin Walters [Wed, 2 May 2018 14:33:32 +0000 (14:33 +0000)]
lib/deploy: Do post-ops when removing staged commit
These are further fixes based on running more of the rpm-ostree
test suite.
When dropping the staged deployment, we do need to do the
"post operations" such as bumping the sysroot mtime, so that
clients know something changed. We also need to regenerate
the deployment refs. And of course do a sysroot reload.
Also, add a "base cleanup" after creating a staged deployment
which also regenerates the refs.
Colin Walters [Mon, 30 Apr 2018 18:41:30 +0000 (14:41 -0400)]
deploy: Don't prune repo at finalization time by default
Doing so can break rpm-ostree, which wants to own the cleanup process
to ensure its baselayer refs are generated.
Further, doing the cleanup at shutdown time adds latency. It's also
going to be generally unnecessary as we expect repo pruning to have
been done when writing the refs.
Jonathan Lebon [Mon, 30 Apr 2018 12:53:01 +0000 (08:53 -0400)]
man/ostree: Document --repo option a bit more
This new information is already mostly part of `ostree.repo(5)`, though
let's put it in `ostree(1)` as well since that's where the switch is
officially documented.
Colin Walters [Mon, 23 Apr 2018 21:53:04 +0000 (17:53 -0400)]
lib/repo: Enable locking by default, but drop external API
The code has been sitting around for a while but since I disabled
it by default, I doubt anyone is really using it or relying on it.
This patch and turns on locking by default, and also drops the
API which was only public in the experimental API builds.
Conceptually these are two distinct things, and we
may actually want to split up the patches.
I don't think this will break anyone, but it's hard to say for sure.
It's also going to be hard to find out until we actually release
I suspect...
But anyone who is broken should be able to add `locking=false` into
their repo config. On the flip side Endless has been shipping with
this enabled and it is reported to help.
The reason to drop the APIs: I'm a bit concerned about the interactions over time
between libostree's use of the API and any apps that start using it.
For example, if an app specifies a SHARED lock in their code, then
later internally we decide to temporarily grab an `EXCLUSIVE`, but the
app had a second thread/process that was `EXCLUSIVE` already, and
that process was waiting on the first bit of code, then we could
deadlock. I can't think of a real world situation where this would happen
yet though.
We are likely to in the future have say `fsck` take an external lock,
`checkout` grab a shared one, etc.
Colin Walters [Fri, 27 Apr 2018 19:33:02 +0000 (15:33 -0400)]
deploy: Return staged deployment
Today rpm-ostree has some code to run a "sanitycheck" on a deployment.
I had initially deleted that when adapting it to use the staging code,
but I realized it should work fine; we just won't see the merged
config, but that's OK.
When I readded that code it started crashing because we didn't
actually return the new deployment object. We'll gain some coverage
here as I'll land the code to have rpm-ostree use staging, then bump
the rpm-ostree tests here.
The fact that `ostree admin deploy` always itself loaded the
merge kargs masked a bug in the core. Let's change our tests
to not pass any kernel arguments to ensure we cover this.
The new logic in the CLI is a bit subtle, but if you read
carefully is a lot clearer I believe. Basically we have one
of a few "starting points" in the first section, which can
then be further augmented.
Colin Walters [Fri, 27 Apr 2018 15:53:43 +0000 (11:53 -0400)]
lib/deploy: Fix staged deployments with no kargs
Testing out the staged API with rpm-ostree, ostree-prepare-root.service
in the initramfs was failing. Turned out that was because we didn't
have a `root=` kernel argument. Which was because we didn't have
any kernel arguments at all except `ostree=`.
That in turn was because we weren't loading the bootloader config
from the merge deployment.
The serialized deployment data holds the unique identity of
(osname, checksum, deployserial) - look for the real merge deployment
in our deployment list which has the bootloader arguments we need.
This issue was entirely masked by the `ostree admin deploy` command
which itself explicitly loads the merge deployment's kernel arguments
in every case - it never passes the `NULL` default down. A followup
patch will fix that.
Jonathan Lebon [Tue, 24 Apr 2018 16:51:38 +0000 (12:51 -0400)]
lib/deploy: Also compare deployment csum versions
When comparing deployments to determine whether we need a new
bootversion, we should also check whether the commit "version" metadata
is the same. Otherwise, we may end up with the a bootconfig whose
`title` includes a version that doesn't match the one from the
deployment checksum.